smartos > Triton Notes

Triton Notes

Created by hww3. Last updated by hww3, 6 years ago. Version #6.

Delegated datasets are the recommended approach; but you have to specify that on zone creation, and later expansion using zfs would be complicated.
About 6 months ago, I set up a mini Triton DataCenter on a Dell R710. Right now I don't have any separate compute nodes, and am provisioning VMs on the head node. Since this is a simple setup in my basement, I don't see the need (at this point) to have a separate headnode for 10 VMs. Overall, I've been really pleased with my Triton experience. A few notes of problems I have run into:

ZFS Disk layout

You don't get any real control over how the zpool is laid out when setup does its thing, and ZFS doesn't really allow you to change the pool layout once it's created.

What I have discovered is this:

1 disk -> concat

2 disks -> mirror

3 disks -> mirror + log

4 disks -> mirror + log + cache

etc

I'm completely fine with adding the log and cache disks later but it would be nice to have the ability to specify how the core disks get used (perhaps an option in the json inputfile?), even if it's just a mirror/raid1z/raid2z/etc designation.

I really wanted a 4 disk raidz pool, and the only way I was able to get there was to start out with a dummy disk then migrate the data to a new zpool set up the way I wanted and then rename it. I ended up wasting a lot of time getting everything set up the way I wanted due to having to do a lot of zfs filesystem rebuilding.

There are plenty of documents online that talk about this, but the general idea is to start with an extra disk that won't ultimately be in your pool (if you're planning to have a log or cache disk, these are a good choice). Let Triton do its thing on installation, then reboot into recovery mode without mounting the disks. Then you can create a new zpool (with a name other than zones, preferably not mounted) with your desired data disk configuration. Once that's done, do a zfs send/receive from the zones pool (you'll need to import the zones pool without mounting it) on the dummy disk onto your new pool. Export the old zones pool (and remove the disk!) then rename the new pool to zones.

If you've done everything right, you should be able to reboot and have the new permanent zones file ready to go.

Update: it seems that you can use the config keys "layout", "cache" and "spares" to set the type of ZFS pool type, whether there is a cache disk and the number of spares. This should be set in /usbkey/config… Reference: sdc-headnode/scripts/joysetup.sh.
 
Update 2: You might also be able to drop to a command prompt and create the zones pool yourself before running the installer to completion.

Network behavior and packet loss

Have had some truly weird network behavior on VNICS(ie external0 vs bnx0): high rates of packet loss (upwards of 25% over time) on a basically quiet network. Traced it down to one of 2 possibilities: having a Netgear R6250 anywhere on that network range regardless of where it was physically connected, or possibly SoftEther VPN. I got rid of the Netgear device and replaced it with a pfsSnse applicance and things were back to good, but had some periodic problems when I was testing SoftSther for remote access. I haven't investigated further but have had identical problems with both the onboard NIC(Broadcom Net Extreme II) and a pci-e adapter (an Intel T310)

DHCP Client woes

Any KVM that needs to perform DHCP requests is probably not going to work. These seem to be filtered. So, for example, using pfSense in a KVM to connect to upstream internet that assigns addresses using DHCP doesn't work. Note: may work if allow_unfiltered_promisc is set; unclear, as KVM has a built-in dhcp server that is used to assign addresses to interfaces.

Running a DHCP server is ok (and well documented), just turn off DHCP spoofing protection. 

Softether

I wanted to run a Softether VPN server for remote access and had a lot of trouble getting it to work. It turns out that Softether isn't completely up-to-date on Solaris networking and isn't able to connect to the virtual nics available in zones. I have a PR submitted to the Softether project that solves this problem (see https://github.com/SoftEtherVPN/SoftEtherVPN/pull/438 )
 
Note that you will probably need to enable all of the spoofing options, including allow_unfiltered_promisc. Note that that last option is currently not retained on reboot, so you'll need to add it manually using dladm set-linkprop -p promisc-filtered=off from the GZ each time you restart the zone running softether. An alternate solution would be to modify /usr/lib/brand/joyent/statechange file to permit setting this property. The change is very simple but requires the platform image to be rebuilt, which is doable, but not for the faint of heart! I have a set of step-by-step instructions for building a platform image; get in touch if you'd like a copy.

NTP

The global zone runs ntpd to synchronize the local clock for all zones on the box, but if you want to run a time server for other clients on your network, the process is a bit tricky. You can't update the clock from non-global zones (and why would you want to anyhow, if it's already being done from the GZ?), and the NTP server package in pkgsrc, openntpd, won't seem to allow clients to synchronize if it doesn't have an upstream time server to synchronoze against. The solution is to use isc ntpd, which is already installed on each zone. You'll need to configure it with a dummy time source since the local clock is already synchronized. The key to doing this looks something like:

# Use the local clock
server 127.127.1.0 prefer
fudge  127.127.1.0 stratum 10

You'll also need to modify the SMF method file, /lib/svc/method/ntp, as it checks to see if the service is being run in the GZ. Remove that check, and you should be able start ntpd and serve network clients (best to verify this before relying on it, though!)

Zone safeguards

It would be really great to be able to lock a vm so that it can't be deleted. I have accidentally deleted a vm because it had a uuid that had a similar prefix to the one I wanted to delete. Something like "vmadm destroy -i UUID" would be great, especially if it printed the respective line from "vmadm list" before prompting.

Update: turns out there is a way to effectively lock a vm: the indestructible_delegated and indestructible_zoneroot options on the vm record. Setting these to true enforces a 2-step delete. Triton has support for setting these; look for references to "deletion protection" when creating a new zone through the triton tools.

Data storage

Finally, I know that this goes against the Triton mantra, but unfortunately there are certain situations where data needs to be preserved as vms that use it are created and destroyed... think very large traditional databases, or filesystems with data for which zfs send and recv would take too much time.

Recognizing that such things tie instances to specific hardware, it would be really great that if I had a disk shelf containing some zpool other than zones on a machine, there were some method to lofi or delegate a dataset.
 
 

Update: there is a filesystems key that can be used to mount filesystems, but it doesn't appear to be updatable, so the zone would have to be created with the data. Alternately, zonecfg could be used to add the filesystem, but its presence would not be reflected in vmadm. 

KVM fanciness

I was able to crib some work done for FreeBSD and created integrated KVM images for NetBSD 6.5.1 and 7.1 that seem to work just fine with all of the SDC bells and whistles. The guest integration is simple and elegant.

The KVM drivers don't seem to work with non-server Windows OS. Obviously, running workstation Windows isn't really the goal of SmartOS/Triton, but there are situations where you might want to do that. Luckily, Redhat's virtio driver disk includes drivers that do work (for virtio and NetKVM)

On "private-cloud" Triton, the windows administrator password doesn't seem to be populated automatically, so you end up with a vm that can't be logged into. One solution: change the setup complete script on the guest tools image to look at root_pw instead of administrator_pw. This seems to work, and has the benefit of creating sufficiently complex passwords that the one-shot password change script doesn't fail to set a password because a manually provided administator_pw value is too simple.

The qemu-extra-opts key can be used to pass flags to the qemu server process. This is useful for specifying a floppy drive image (required for the virtio drivers under XP) and so forth. Just remember to unset them after use so that you don't have a floppy drive perpetually mounted up. 

Not categorized | RSS Feed | BackLinks